Simulating tracking shots from image sequences

ABSTRACT

A simulated tracking shot is generated from an image sequence in which a foreground feature moves relative to a background during capturing of the image sequence. The background is artificially blurred in the simulated tracking shot in a spatially-invariant manner corresponding to foreground motion relative to the background during a time span of the image sequence. The foreground feature can be substantially unblurred relative to a reference image selected from the image sequence. A system to generate the simulated tracking shot can be configured to derive spatially invariant blur kernels for a background portion by reconstructing or estimating a 3-D space of the captured scene, placing virtual cameras along a foreground trajectory in the 3-D space, and projecting 3-D background points on to the virtual cameras.

TECHNICAL FIELD

The subject matter disclosed herein generally relates to imageprocessing and editing. More particularly, the subject matter relates tomethods and systems for automated generation of a simulated still imagefrom an image sequence, such as a video clip.

BACKGROUND

Tracking is a photographic technique often used to take a photograph orpicture of a moving subject such that a sense of the subject's motion iscaptured. Tracking shots can be achieved by moving the camera andkeeping the subject in more or less the same position of the frame forthe duration of the exposure. When executed correctly, this techniqueresults in a photograph when the moving foreground is relatively sharp,while the background is blurred consistent with the camera's motionduring exposure.

Satisfactory tracking shots, however, can be difficult to capture,particularly for novices and less experienced photographers. Thechallenges inherent in capturing a good tracking shots are amplified incases where the motion of the subject is complex, such as where thesubject follows a nonlinear path.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments of the disclosure are illustrated by way of example,and not by way of limitation, in the figures of the accompanyingdrawings.

FIG. 1 is a schematic diagram of a system for generating a simulatedtracking shot, in accordance with an example embodiment.

FIG. 2 is a schematic diagram of a blur derivation module which may formpart of the system of FIG. 1, in accordance with one example embodiment,the blur derivation module being configured to derive blur parametersfor tracking shot simulation from an input image sequence.

FIG. 3 is a flowchart showing an example embodiment of a method forgenerating a simulated tracking shot based on an image sequence, inaccordance with an example embodiment.

FIG. 4A is a schematic diagram showing example image processing outputsduring processing of an example image sequence in accordance with theexample method of FIG. 3.

FIG. 4B is a schematic diagram of a reconstructed 3-D spacerepresentative of a scene captured in the example image sequence of FIG.4A, the reconstructed 3-D space including a set of virtual cameras forgenerating spatially varying blur kernels in accordance with an exampleembodiment.

FIG. 5 is a schematic diagram illustrating that dynamic points in 2-Dcan be mapped to multiple trajectories in 3-D.

FIG. 6 is a schematic representation of an example sparse blur kernelmap derived from the example reconstructed 3-D space of FIG. 4B, andoverlaid on an example reference image selected from the example imagesequence of FIG. 4A.

FIG. 7 is an example embodiment of a simulated tracking shot generatedfrom the image sequence of FIG. 4A, in accordance with one exampleembodiment.

FIG. 8 is a diagrammatic flowchart of a method for estimating depthsvalues for respective background points in an image sequence, inaccordance with an example embodiment.

FIG. 9 is a block diagram illustrating components of a machine,according to some example embodiments, configured to read instructionsfrom a machine-readable medium and perform any one or more of themethodologies described herein.

DETAILED DESCRIPTION

In the following description, for purposes of explanation, numerousspecific details are set forth to provide a thorough understanding ofexample embodiments. It will be evident to one skilled in the art,however, that the present subject matter may be practiced without thesespecific details.

Overview

One aspect of the disclosure provides a method comprising generating asimulated tracking shot based on an image sequence. The image sequencemay form part of a video clip, having been captured by a video camera.In other embodiments, the image sequence may comprise a series of stillimages (e.g., digital photographs or pictures) captured by a camera.

A “tracking shot” means a still image having motion-blurred backgroundfeatures and a foreground feature which is substantially withoutmotion-blurring, the motion-blurring of the background featurescorresponding to camera movement or foreground movement relative to therespective background features. A video clip of a subject scenecomprising the foreground feature moving relative to the background maythus be processed to produce a tracking shot that simulates trackingmovement of the foreground feature relative to the background with anexposure time corresponding to several frames of the video clip.

In an example embodiment, a photo editing application (e.g., Adobe®Photoshop®, LightRoom® or the like) may derive spatially variant blurparameters for at least part of a background portion of a referenceimage selected from the image sequence. The simulated tracking shot maythen be generated by modifying the reference image by blurring thebackground portion of the reference image based on the derived spatiallyvarying blur parameters. The spatially invariant blurring of thebackground portion thus reflects relative movement of the backgroundduring a time span of the image sequence (corresponding to an exposuretime of the simulated tracking shot), with the foreground portion of thesimulated tracking shot being substantially unblurred relative to theforeground portion of the reference image.

In some embodiments, the selecting of the reference image may compriseuser selection of the reference image. The system may thus be configuredto prompt a user to select the reference image, for example includingdisplaying respective images in the image sequence. In otherembodiments, the reference image may be selected automatically, e.g., byan image in the middle of the time span.

“Spatially varying” parameters, values, or variables means that therelevant parameters, values, or variables are not uniform for an entireimage, but may vary between different positions in a two-dimensionalimage. The spatially variant blur parameters may comprise spatiallyvarying blur kernels (e.g., per-pixel blur kernels for at least thebackground portion) for the reference image. As is well-established, akernel is a matrix which is small relative to a target image, and whichcan be used in image processing for blurring, sharpening, embossing,edge detection, and the like, by means of convolution between the kerneland the target image. A “blur kernel” means a kernel which is configuredfor blurring the target image. In this instance, the blur kernels may beconfigured for applying motion blurring to the background portion, withdifferent parts of the background portion being blurred differently,based on their respective motions relative to the image foregroundduring the image sequence.

In such case, generation of the modified image may comprise convolvingthe reference image and the per-pixel blur kernels. The method maycomprise associating dotted blur kernels with respective image pixels inthe foreground feature. A “dotted blur kernel” means a blur kernel thatapplies no blurring to the corresponding image portion.

The method may comprise reconstructing a three-dimensional space of thesubject scene, and may include determining a foreground trajectory thatreflects movement of the foreground feature in the reconstructedthree-dimensional space. The reconstructed three-dimensional space ofthe subject scene may include multiple three-dimensional backgroundpoints corresponding to the background portion of the reference image.In one embodiment, reconstruction of the three-dimensional spacecomprises automated Structure-from-Motion (SFM) analysis. In anotherembodiment, reconstruction of the three-dimensional space may compriseestimating relative depths for the multiple background points based onanalysis of feature tracks that track movement of multiple backgroundfeatures over the time span of the image sequence.

Derivation of the blur kernels may comprise positioning a plurality ofvirtual cameras at respective three-dimensional positions in thereconstructed three-dimensional space, and projecting the multiplebackground points on to the plurality of virtual cameras. Eachbackground point thus produces a two-dimensional projection point oneach of the plurality of virtual cameras, and thereby providing a set oftwo-dimensional projection points. The set of two-dimensional projectionpoints for each background point may be combined or collated to producea corresponding blur kernel, or to produce blur parameters from which acorresponding blur kernel may be derived.

The plurality of virtual cameras may be placed at time-indexed positionsalong the foreground trajectory. In another embodiment, positions forthe virtual cameras along the foreground trajectory may be determined byequally sampling several positions on the foreground trajectory around atime instance corresponding to the reference image. In one embodiment,the plurality of virtual cameras have a common, fixed rotation in thethree-dimensional space and are centered on the foreground trajectory.In such case, the foreground feature is located centrally in thetwo-dimensional image plane of each virtual camera.

More particular features of example embodiments of the disclosed methodand system will now be described.

Example Embodiment

As described herein, in some example embodiments, systems and methodsare described that are configured to process image sequences and/orcreate modified or simulated images via an image or photo editingapplication, such as the Adobe® Photoshop® family of applications. Thetechnology may be implemented by one or more applications resident on acomputing device (e.g., mobile computing device) and/or in a networkedenvironment (e.g., a cloud-based network environment) where processingmay, or may not, be distributed.

FIG. 1 is a block diagram illustrating a system 100, in accordance withan example embodiment, for simulating a tracking shot based on an imagesequence. The system 100 may be implemented on any number of differenthardware components. In an example embodiment, the system 100 isimplemented on a computer system having one or more processors toexecute respective operations of the method. In another exampleembodiment, the system 100 may be executed using a graphics processingunit (GPU).

In some embodiments, the system 100 may reside on a user machine, suchas a personal computer (PC), tablet computer, smartphone, and the like.In other embodiments, the system 100 may reside on a centralized serverin communication with a user machine. In such embodiments, the usermachine may direct the centralized server to perform one or moreoperations of a process for generating a simulated tracking shot, forexample to reconstruct a three-dimensional space of a subject scene andto derive spatially varying blur kernels for a background portion. Itshould be noted that hybrid embodiments are also envisioned, where someaspects of the generation of a simulated tracking shot are performed ona centralized server. This allows a user machine to offload part or allof the image analysis and/or simulated image generation to anothermachine for faster processing.

The system 100 may include an image data module 104 to receive andaccess image data for an image sequence of a subject scene. In thisexample, the image data comprises an input video clip 103 selected by auser through operation of the image data module 104. The system 100 alsocomprises a reference image module 108 for selecting a reference imagefrom the inputted image sequence. In this example, the reference imageis selected by the user, and is provided by a particular selected frameof the input video clip 103.

A blur derivation module 112 is configured to process the image data toderive spatially variant blur parameters (e.g., spatially varying blurkernels) for at least parts of the background portion of the referenceimage. The spatially variant blur parameters are representative ofmotion of respective parts of the background of the subject scenecaptured in the video clip 103 relative to a foreground portion ortracking subject of the video clip 103. The system 100 further includesa simulation generator module 116 to generate a modified image, relativeto the reference image, that represents a simulated tracking shot 117based on the input video clip 103. The background portion of thesimulated tracking shot 117 is thus blurred to simulate the relativemovement of the background during a time span of the image sequence (or,in some cases, a smaller time span within the image sequence), while aforeground portion of the simulated tracking shot 117 is substantiallyunblurred relative to the reference image.

FIG. 2 is a block diagram showing an example embodiment of the blurderivation module 112. The blur derivation module 112 includes an imageanalysis module 205 to process the image data provided by the inputvideo clip 103 to enable reconstruction or estimation of athree-dimensional space within which the subject scene is set. The imageanalysis module 205 may thus be configured to provide as output areconstructed 3-D space for the subject scene of the input video clip103. Such a reconstructed 3-D space may comprise 3-D coordinates for atleast some background features forming part of the background portion.

The image analysis module 205 may further include a segmentation module210 for processing the image data to automatically identify theforeground portion of the reference image, and to segment the foregroundportion and the background portion. A motion analysis module 215 mayfurther be provided for determining from the image data a foregroundtrajectory that reflects movement of features forming part of theforeground portion relative to the background. The foreground trajectorymay be a three-dimensional trajectory in the reconstructed 3-D space.

The blur derivation module 112 may further include a virtual cameramodule 220 to position a plurality of virtual cameras at respectivepositions in the three-dimensional space of the subject scene. In thisexample embodiment, the virtual camera module 220 is configured toposition the plurality of virtual cameras along the foregroundtrajectory. A background projection module 225 is provided to projectmultiple background points in the reconstructed three-dimensional spaceon to the plurality of virtual cameras, to provide a two-dimensionalprojection point on each of the plurality of virtual cameras for each ofthe multiple background points. The blur derivation module 112 mayfurther include a blur kernel module 230 that is configured to derive aseparate blur kernel for each of the multiple background points based ona corresponding set of two-dimensional projection points on theplurality of virtual cameras.

Further details of examples operation of the respective modules brieflydescribed above will become evident from the following description of anexample embodiment of a method of generating a simulated tracking shotsfrom an image sequence, as illustrated schematically by the flow diagramin FIG. 3. The method 300 may be performed by use of the example system100.

At operation 303, digital image data comprising an input image sequenceis received from a user. As mentioned, the input image sequence is, inthis example embodiment, an input video clip 103. In other embodiments,a sequence of input images can comprise a series of still images, suchas a series of digital photographs captured with a photo camera. Acommon feature of more sophisticated digital cameras is thefunctionality of capturing a series of closely spaced digital pictures,often referred to the so-called burst mode, in which the cameracontinues capturing different digital pictures for as long as a cameratrigger is depressed by the user. The digital pictures in such imagesequences are typically taken at relatively high shutter speeds, so thata background of the respective still images is captured in sharp focus.This typically fails to convey to a viewer in a sense of motion of theforeground feature relative to the background. It is a benefit of thedescribed functionality of generating a simulated tracking shot based onsuch sharply focused action images, that it enables the provision of atracking shot in which the foreground is in relatively sharp focus,while motion of the foreground or subject of the simulated tracking shotis indicated by accurately simulated blurring of the background.

FIG. 4A shows a schematic representation of the example input video clip103, consisting of a sequence of digital images captured at different,sequential times in respective frames 404 of the video clip 103. Thevideo clip frames 404 are of a common subject scene, in this exampleembodiment capturing movement of a hiker 408 through an outdoorlandscape. Because the hiker 408 is the subject of tracking focus in thevideo clip 103, the hiker 408 in this example embodiment provides aforeground of the subject scene, while features of the outdoor landscapeform part of the background of the subject scene. Note that thedistinction between background and foreground features is not basedstrictly on real-world three-dimensional depth of the respective partsof the scene, but part of at least some parts of the background can belocated closer to the camera than the foreground features. In thisexample, parts of the path along which the hiker 408 walks during a timespan of the video clip 103 are closer to the camera, but are backgroundfeatures relative to which the hiker 408 moves.

At operation 306, a reference image 412 (see FIG. 4A) is selected fromthe video clip 103. In this example embodiment, the reference image 412is selected by the user, and is provided by a selected one of the videoframes 404. This operation may comprise prompting the user to select oneof the video frames 404, and may also include provision of a navigationinterface on a user device, to enable the user to page or scan throughthe video frames 404. Selection of the reference image 412 may in suchcase comprise, e.g., left clicking by the user on a user-preferred frame404.

At operation 309, the image data of the video clip 103 (in this exampledigital video data) is processed to automatically segment the foregroundand the background of the subject scene which is captured in the videoclip 103. In some embodiments, the operation of segmenting of theforeground and the background may include receiving foreground featureidentification input or assistance from the user, for example byallowing the user to paint a target foreground feature on a userinterface of the image processing application. In this exampleembodiment, however, foreground and background segmentation is fullyautomatic, being performed without any user input. An output of thesegmentation operation, at operation 309, includes a motion segmentationmask 416 that defines a foreground portion 420 of the image provided bythe relevant frame 404, and including the relevant foreground features.In this example, the motion segmentation mask 416 thus contiguouslycovers the hiker 408 which is to be the focus feature of the simulatedtracking shot 117. The remainder of image frame 404 (the inverse of theforeground portion 420 provided by the motion segmentation mask 416)forms a background portion 424 of the frame 404, and encompasses variousbackground features of the scene captured in the video clip 103.

Various methods for automated motion segmentation of dynamic scenescaptured on video are known. In this context, dynamic scenes consist ofmultiple moving objects with a static or moving camera. The objective ofmotion segmentation is in these instances to segment featuretrajectories according to respective motion in the scene. Existingmotion segmentation techniques can be divided roughly into feature-basedor dense-based methods. In feature-based methods, objects or featuresare represented by a limited number of points (e.g., corners or salientpoints), while dense methods compute a pixel-wise motion (see, forexample, Kumar et al., “Learning layered motion segmentations of video”,International Journal of Computer Vision, vol. 76, 2008). In thisexample embodiment, the automated segmentation, at operation 309, is adense method, comprising computing optical flow between neighboringframes 404 of the video clip 103, and extracting the motion segmentationmask 416 based on the optical flow information thus produced.

At operation 312, 3-D reconstruction of the scene is performed, thusreconstructing a virtual 3-D space (indicated schematically by referencenumeral 427 in FIGS. 4A and 4B) by estimating or determining 3-Dcoordinates of various features and/or portions of the subject scenerepresented in the video clip 103. The 3-D reconstruction, at operation312, recovers 3-D camera motion and a sparse set of 3-D static scenepoints, or background points 435 (see FIG. 4B), using astructure-from-motion (SFM) process. In this context, “sparse” meansthat the density of 3-D points is lower than the pixel density of theprimitive image provided by the relevant frame 404. SFM is a well-knowntechnique for deriving 3-D information from dynamic video footage, andcommercial 3-D camera trackers like Boujou™ and Syntheyes™ are commonlyused. In this example embodiment, the Voodoo™ camera tracker applicationis used for 3-D reconstruction.

At operation 315, information from the motion segmentation operation (atoperation 309) and the 3-D space reconstruction (at operation 312) areused in combination to determine a foreground trajectory 429 (FIG. 4A)that reflects movement of the foreground feature (in this example thehiker 408) in the reconstructed 3-D space 427. It will be appreciatedthat, because of the foreground move steering the time span of the videoclip 103, SFM processes that deal with a static scene will typically beinsufficient to recover the foreground trajectory 429. Nonrigidstructure-from-motion (NRSFM) methods can, however, be used toreconstruct nonrigid scenes from a monocular video. Existing NRSFMmethods include recovery of the nonrigid structure from motionfactorization (see, for example, Dai et al., “A simple prior-free methodfor nonrigid structure from motion factorization”, in CVPR, 2012) or byconstrained linear solutions (see, for example, Zhang et al., “Motionimitation with a handheld camera”, IEEE Transactions on Visualizationand Computer Graphics, vol. 17, 2011, or Park et al., “3d reconstructionof a moving point from a series of 2d projections”, in ECCV, 2010). Anyof these processes may be used for the purposes of operation 315.

The NRSFM process used in this example embodiment for foreground motionrecovery will now be described. For a static scene point, 3-Dreconstruction can be calculated based on rays connecting each framelocation to its corresponding camera center intersect at the true 3-Dlocation of the point. The process is known as triangulation. Thetriangulation constraint does not, however, apply when the point movesin the duration between frames. In these cases NRSFM reconstruction maybe employed.

FIG. 5 shows an example, where the captured 2D point u_(i) cancorrespond to multiple 3D points X_(i) or X_(i)′. Each of the 2-D pointsgives a constraint

u _(i) =P _(i) X _(i)

where P_(i) is the corresponding projection ray of the point on to theimage plane. In order to make the solution well constrained, severaltemporal constraints proposed in Zhang, et al., are adopted, as follows:

E=Σ _(i)(P _(i) X _(i) −u _(i))²+αΣ_(i)(X _(i) −X _(i−1))²+βΣ_(i)(2x_(i) −x _(i−1) −X _(i+1))²  (1)

where the first-term minimizes the fitting error. Coefficients α and βare customizable. The second and third terms provide regularization ofthe unknowns, which minimize the first and second derivativesrespectively. The regularization is reasonable, since the moving objectin general does not change abruptly in consecutive frames. Equation 1 isa quadratic energy function and has a closed form solution. This methodprovides the foreground trajectory 429 illustrated schematically inFIGS. 4A and 4B.

At operation 318, the reconstructed 3-D space 427 and the motionsegmentation mask 416 remove the foreground information from thereconstructed 3-D space 427, leaving multiple background points 435 thattogether provide a sparse set of 3-D coordinates for respective pointsforming part of the background portion 424.

At operation 321, spatially varying blur parameters are derived based onthe image data, in this example comprising generating spatially varyingblur kernels. This comprises, at operation 324, positioning a pluralityof virtual cameras 433 (see FIG. 4B) along the dynamic foregroundtrajectory 429 in the reconstructed 3-D space 427, and then, atoperation 327, projecting the set of sparse 3-D background points 435 onto the virtual cameras 433. Each 3-D background point 435 thus producesa corresponding 2-D projection point on each of the virtual cameras 433.Thereafter, at operation 330, a respective blur kernel 606 (see, e.g.,FIG. 6) is determined or synthesized for each background point based oncombining or collating the corresponding set of 2-D projection points onthe virtual cameras 433. The derived blur parameters thus comprise, inthis example embodiment, a sparse set of spatially varying blur kernels606. Some aspects of operation 321 will now be described in greaterdepth.

To imitate a tracking procedure with respect to the foreground feature(e.g., the hiker 408) the virtual cameras 433 are, in operation 324,centered on the foreground trajectory 429. The series of virtual cameras433 thus simulates a time-sequenced, moving image plane in which aprojection of the foreground feature (e.g., hiker 408) remainsconstantly centered in the image frame. The virtual cameras 433 have afixed, common rotation within the reconstructed 3-D space 427, so thatthe series of virtual cameras 433 simulates only translation, but notrotation, of a camera through the 3-D space 427.

As is well-established in the art, a virtual camera may be modeledmathematically by an intrinsic matrix K_(v), a rotation matrix R_(v) anda camera center C_(v). A projection matrix P_(v) can then be composed asP_(v)=K_(v)[R_(v)|−C_(v)]. For further details in this regard, seeHartley, et al., “Multiple View Geometry in Computer Vision”, vol. 2,Cambridge Univ Press, 2000. From the 3D reconstruction (at operation312), 3-D camera motion is obtained for each of a number of the videoframes 404 of the video clip 103. The 3-D camera motion also consists,for each relevant frame 404, of intrinsics K_(r), rotations R_(r) andcamera centers C_(r). In some embodiments, a virtual camera 433 can beprovided for each image in the primitive image sequence. In thisexample, however, a subset 403 of video frames 404, spaced at regularintervals, is selected from the video clip 103. The series of virtualcameras 433 thus consists, in this example, of 10-15 video frames 404,even though the video clip 103 can comprise hundreds of frames 404.

The intrinsic is borrowed for the virtual cameras 433, K_(v)=K_(r), and,as mentioned, virtual camera rotations are set to a fixed rotationcorresponding to the reference image 412 at time t as R_(v)=R^(t) _(r).The camera centers C_(v) are set to respective positions on the dynamicforeground trajectory 429. In this example embodiment, several positionon the foreground trajectory 429 are equally sampled around time t (e.g.consisting of five positions before time t, and five positions aftertime t).

As mentioned briefly above, the blur kernels 606 are generated byprojecting the three-dimensional background points 435 on to the virtualcameras 433. The projection of one 3-D point onto one virtual camera 433gives a single 2-D projection point. Projection of one 3-D point on tothe series of virtual cameras 433 thus gives a set of 2-D positions fromwhich, in this example, is taken one blur kernel 606 corresponding tothe 3-D background point from which it was projected. In otherembodiments, some processing of the set of 2-D projection points may beperformed to synthesize a blur kernel, for example by connecting the 2-Dprojection points in instances where there are discontinuities in theset of 2-D projection points. FIG. 6 shows an example of a set of sparsespatially varying blur kernels 606 generated for the reference image 412from the video clip 103. Because the blur kernels 606 are generateddirectly by projection of 3-D background points 435, the blur kernels606 are physically correct. A depth of respective point on thebackground portion 424 of the reference image 412 can, for example,readily be observed from the length of the corresponding blur kernel606.

At operation 333 (FIG. 3), a set of dotted blur kernels 616 (FIG. 6)coincident with the foreground portion 420 is added to the spatiallyvarying blur kernels 606 of the background portion 424. This preventsblurring of the foreground portion 420 during the blurring process,leaving the foreground portion 420 substantially unblurred, as ischaracteristic of a well-executed tracking shot. The set of dotted blurkernels 616 is added on the foreground portion 420 based on thesegmentation mask 416 obtained in motion segmentation operation 309.

At operation 336, the respective blur kernel 606 is calculated for eachimage pixel in the reference image 412, to provide a set of blur kernels606 for the reference image 412 at per-pixel density. In this exampleembodiment, a simple interpolation method based on relative distances isused for the interpolation process. In other embodiments, more advancededge-aware interpolation techniques can be used.

Finally, at operation 339, the reference image 412 and the per-pixeldense blur kernels 606 of them convoluted to generate the simulatedtracking shot 417, an example of which is shown in FIG. 7. As can beseen, the hiker 408 (which provides the foreground feature in thisexample) in the simulated tracking shot 417 is substantially unblurredfrom the reference image 412, while the background is blurred in amanner that simulates motion blur caused by relative movement betweenthe camera and the respective background features with a slow shutterspeed. In some embodiments, the image processing application may beconfigured to present the user with blur modification options to adjustthe amount of blur applied to the background. The user may, for example,adjust a blur parameter according to which the spatially varying blurkernels 606 may be scaled up or down, as the case may be.

The simulated tracking shot 417 generated by the method described withreference to FIG. 3, which includes 3-D reconstruction, can beconsidered as a physically-correct shot, because it is a simulation ofreal capture procedure. In other embodiments, a less rigorous techniquemay be used for estimating depths of respective background points 435.This less rigorous technique is referred to herein as a 2.5-D method, anexample of which is described in what follows.

A 3-D reconstruction delivers 3-D coordinates for respective backgroundfeatures and/or scene points, and thus provides absolute depth values. Acorrect depth-aware tracking shot can, however, be synthesized based onrelative depth information between features in the image scene. To thisend, relative depth is extracted from feature tracks of a few frames404.

Consider one feature track, by which is meant a track followed by aparticular feature in the scene over the course of the considered frames404. An average coordinate difference of a single feature track isdenoted P_(i). If there is no camera rotation, the features that arecloser to the camera should have larger values for P_(i). A comparisonof respective feature track information may thus be used to estimaterelative depths of different parts or features of the scene. FIG. 8shows an example embodiment of a method 800 to estimate background depthinformation, to enable tracking shot simulation based in a 2.5-Dprocess. Broadly, the method 800 comprises tracking movement over thetime span of the image sequence (e.g., the video clip 103) of multiplebackground features forming part of the background portion, to determinerespective feature tracks for the multiple background features. Relativedepth information for multiple background points in the backgroundportion 424 is then estimated based on the respective identified-featuretracks.

As will be described below, the depth estimation comprises anoptimization procedure for each feature track with reference toneighboring feature tracks. Relying on spatial distance between featuretracks to identify neighbor features can, however, be problematic incertain instances, as there could exist depth discontinuity where twopoints are spatially close in the image, but are far away from eachother in depth. To account for such discontinuities, over segmentationis applied, at operation 808, to segment the frame 404 into multiplesuper pixels, as proposed by Felzenszwalb, et al., in “Efficientgraph-based image segmentation”, International Journal of ComputerVision, vol. 59.

Video stabilization is applied before computing the average coordinatedifference (P_(i)) for a feature track. In this embodiment, the videostabilization as applied to remove camera rotation, so that featuretrack scale differences are due only to camera translation. In otherembodiments, (e.g., where camera motion during video capture is apanning operation in which the camera swivels to follow a foregroundfeature moving substantially across the image plane) video stabilizationmay comprise removing translation, rather than removing rotation. Foreach feature track, a scale value α_(i) is assigned by solving thefollowing energy equation (Equation 2):

$E = {\min\limits_{\alpha}{\sum\limits_{i}\; \left( {{{{\alpha_{i}\overset{\_}{P}} - \left. 〚{S\left( P〛 \right.}_{i} \right)}} + {\sum\limits_{j \in N_{i}}\; {{\alpha_{i} - \alpha_{j}}}}} \right)}}$

wherein

-   -   P_(i) is the average coordinate difference of track i,    -   P is the average of all P_(i)    -   S(.) is a video stabilization operator, and    -   N_(i) refers to the neighboring tracks of i.

Optimization based on the above equation, to provide respective scalevalues for the feature tracks, is based on the insight that neighboringtracks (j) of track (i) should have similar scale values (α). To accountfor depth discontinuities, spatial distances are computed only forfeatures within the same super pixel. Equation 2 is quadratic and canthus be solved by a least-square regression procedure.

At operation 816, video stabilization and optimization of feature trackscale values for the respective features are performed according toEquation 2. This provides a scale value α for each feature identifiedfor tracking. At operation 824, the scale values are mapped to the 3-Dspace by an exponential function, to provide a depth value for each ofthe features. In this embodiment, 2^(∝) ^(i) is applied empirically.

In this manner, background depths for multiple points in the backgroundportion 424 are provided, without performing rigorous 3-D reconstructionof the image scene. These estimated background depths can be used forprojection on to a plurality of virtual cameras 433, to derive blurkernels for simulated tracking shot generation.

Another embodiment of a method for generating a simulated tracking shotfrom video footage or an image sequence is thus similar to the method300 described with reference to FIG. 3, with the exception that theoperations of reconstructing the 3-D space (at operation 312) anddetermining 3-D background depths (at operation 318) are insteadreplaced by depth parameter estimation consistent with or analogous tothe method 800 described with reference to FIG. 8.

FIG. 9 is a block diagram illustrating components of a machine 900,according to some example embodiments, able to read instructions from amachine-readable medium (e.g., a machine-readable storage medium) andperform any one or more of the methodologies discussed herein, in wholeor in part. Specifically, FIG. 9 shows a diagrammatic representation ofthe machine 900 in the example form of a computer system and withinwhich instructions 924 (e.g., software, a program, an application, anapplet, an app, or other executable code) for causing the machine 900 toperform any one or more of the methodologies discussed herein may beexecuted. In alternative embodiments, the machine 900 operates as astand-alone device or may be connected (e.g., networked) to othermachines. In a networked deployment, the machine 900 may operate in thecapacity of a server machine or a client machine in a server-clientnetwork environment, or as a peer machine in a peer-to-peer (ordistributed) network environment. The machine 900 may be a servercomputer, a client computer, a personal computer (PC), a tabletcomputer, a laptop computer, a netbook, a set-top box (STB), a personaldigital assistant (PDA), a cellular telephone, a smartphone, a webappliance, a network router, a network switch, a network bridge, or anymachine capable of executing the instructions 924, sequentially orotherwise, that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include a collection of machines that individually orjointly execute the instructions 924 to perform any one or more of themethodologies discussed herein.

The machine 900 includes a processor 902 (e.g., a central processingunit (CPU), a graphics processing unit (GPU), a digital signal processor(DSP), an application specific integrated circuit (ASIC), aradio-frequency integrated circuit (RFIC), or any suitable combinationthereof), a main memory 904, and a static memory 906, which areconfigured to communicate with each other via a bus 908. The machine 900may further include a video display 910 (e.g., a plasma display panel(PDP), a light emitting diode (LED) display, a liquid crystal display(LCD), a projector, or a cathode ray tube (CRT)). The machine 900 mayalso include an alphanumeric input device 912 (e.g., a keyboard), acursor control device 914 (e.g., a mouse, a touchpad, a trackball, ajoystick, a motion sensor, or other pointing instrument), a storage unit916, a signal generation device 918 (e.g., a speaker), and a networkinterface device 920.

The storage unit 916 includes a machine-readable medium 922 (alsoreferred to as “computer-readable medium”) on which is stored theinstructions 924 embodying any one or more of the methodologies orfunctions described herein. The instructions 924 may also reside,completely or at least partially, within the main memory 904, within theprocessor 902 (e.g., within the processor's cache memory), or both,during execution thereof by the machine 900. Accordingly, the mainmemory 904 and the processor 902 may be considered as machine-readablemedia. The instructions 924 may be transmitted or received over anetwork 926 via the network interface device 920.

As used herein, the term “memory” refers to a machine-readable mediumable to store data temporarily or permanently and may be taken toinclude, but not be limited to, random-access memory (RAM), read-onlymemory (ROM), buffer memory, flash memory, and cache memory. While themachine-readable medium 922 (or computer-readable medium) is shown in anexample embodiment to be a single medium, the terms “machine-readablemedium” and “computer-readable medium” should be taken to include asingle medium or multiple media (e.g., a centralized or distributeddatabase, or associated caches and servers) able to store instructions924. The terms “machine-readable medium” and “computer-readable medium”shall also be taken to include any medium, or combination of multiplemedia, that is capable of storing instructions (e.g., instructions 924)for execution by a machine or computer (e.g., machine 900), such thatthe instructions, when executed by one or more processors of the machineor computer (e.g., processor 902), cause the machine or computer toperform any one or more of the methodologies described herein.Accordingly, a “machine-readable medium” refers to a single storageapparatus or device, as well as “cloud-based” storage systems or storagenetworks that include multiple storage apparatuses or devices. The term“machine-readable medium” shall accordingly be taken to include, but notbe limited to, one or more data repositories in the form of asolid-state memory, an optical medium, a magnetic medium, or anysuitable combination thereof.

Throughout this specification, plural instances may implementcomponents, operations, or structures described as a single instance.Although individual operations of one or more methods are illustratedand described as separate operations, one or more of the individualoperations may be performed concurrently, and nothing requires that theoperations be performed in the order illustrated. Structures andfunctionality presented as separate components in example configurationsmay be implemented as a combined structure or component. Similarly,structures and functionality presented as a single component may beimplemented as separate components. These and other variations,modifications, additions, and improvements fall within the scope of thesubject matter herein.

Certain embodiments are described herein as including logic or a numberof components, modules, or mechanisms. Modules may constitute eithersoftware modules (e.g., code embodied on a machine-readable medium or ina transmission signal) or hardware modules. A “hardware module” is atangible unit capable of performing certain operations and may beconfigured or arranged in a certain physical manner. In various exampleembodiments, one or more computer systems (e.g., a standalone computersystem, a client computer system, or a server computer system) or one ormore hardware modules of a computer system (e.g., a processor or a groupof processors) may be configured by software (e.g., an application orapplication portion) as a hardware module that operates to performcertain operations as described herein.

In some example embodiments, a hardware module may be implementedmechanically, electronically, or any suitable combination thereof. Forexample, a hardware module may include dedicated circuitry or logic thatis permanently configured to perform certain operations. For example, ahardware module may be a special-purpose processor, such as a fieldprogrammable gate array (FPGA) or an ASIC. A hardware module may alsoinclude programmable logic or circuitry that is temporarily configuredby software to perform certain operations. For example, a hardwaremodule may include software encompassed within a general-purposeprocessor or other programmable processor. It will be appreciated thatthe decision to implement a hardware module mechanically, in dedicatedand permanently configured circuitry, or in temporarily configuredcircuitry (e.g., configured by software) may be driven by cost and timeconsiderations.

Accordingly, the phrase “hardware module” should be understood toencompass a tangible entity, be that an entity that is physicallyconstructed, permanently configured (e.g., hardwired), or temporarilyconfigured (e.g., programmed) to operate in a certain manner or toperform certain operations described herein. As used herein,“hardware-implemented module” refers to a hardware module. Consideringembodiments in which hardware modules are temporarily configured (e.g.,programmed), each of the hardware modules need not be configured orinstantiated at any one instance in time. For example, where a hardwaremodule comprises a general-purpose processor configured by software tobecome a special-purpose processor, the general-purpose processor may beconfigured as respectively different special-purpose processors (e.g.,comprising different hardware modules) at different times. Software mayaccordingly configure a processor, for example, to constitute aparticular hardware module at one instance of time and to constitute adifferent hardware module at a different instance of time.

Hardware modules can provide information to, and receive informationfrom, other hardware modules. Accordingly, the described hardwaremodules may be regarded as being communicatively coupled. Where multiplehardware modules exist contemporaneously, communications may be achievedthrough signal transmission (e.g., over appropriate circuits and buses)between or among two or more of the hardware modules. In embodiments inwhich multiple hardware modules are configured or instantiated atdifferent times, communications between such hardware modules may beachieved, for example, through the storage and retrieval of informationin memory structures to which the multiple hardware modules have access.For example, one hardware module may perform an operation and store theoutput of that operation in a memory device to which it iscommunicatively coupled. A further hardware module may then, at a latertime, access the memory device to retrieve and process the storedoutput. Hardware modules may also initiate communications with input oroutput devices, and can operate on a resource (e.g., a collection ofinformation).

The various operations of example methods described herein may beperformed, at least partially, by one or more processors that aretemporarily configured (e.g., by software) or permanently configured toperform the relevant operations. Whether temporarily or permanentlyconfigured, such processors may constitute processor-implemented modulesthat operate to perform one or more operations or functions describedherein. As used herein, “processor-implemented module” refers to ahardware module implemented using one or more processors.

Similarly, the methods described herein may be at least partiallyprocessor-implemented, with a processor being an example of hardware.For example, at least some of the operations of a method may beperformed by one or more processors or processor-implemented modules.Moreover, the one or more processors may also operate to supportperformance of the relevant operations in a “cloud computing”environment or as a “software as a service” (SaaS). For example, atleast some of the operations may be performed by a group of computers(as examples of machines including processors), with these operationsbeing accessible via a network (e.g., the Internet) and via one or moreappropriate interfaces (e.g., an API).

The performance of certain of the operations may be distributed amongthe one or more processors, not only residing within a single machine,but deployed across a number of machines. In some example embodiments,the one or more processors or processor-implemented modules may belocated in a single geographic location (e.g., within a homeenvironment, an office environment, or a server farm). In other exampleembodiments, the one or more processors or processor-implemented modulesmay be distributed across a number of geographic locations.

Some portions of the subject matter discussed herein may be presented interms of algorithms or symbolic representations of operations on datastored as bits or binary digital signals within a machine memory (e.g.,a computer memory). Such algorithms or symbolic representations areexamples of techniques used by those of ordinary skill in the dataprocessing arts to convey the substance of their work to others skilledin the art. As used herein, an “algorithm” is a self-consistent sequenceof operations or similar processing leading to a desired result. In thiscontext, algorithms and operations involve physical manipulation ofphysical quantities. Typically, but not necessarily, such quantities maytake the form of electrical, magnetic, or optical signals capable ofbeing stored, accessed, transferred, combined, compared, or otherwisemanipulated by a machine. It is convenient at times, principally forreasons of common usage, to refer to such signals using words such as“data,” “content,” “bits,” “values,” “elements,” “symbols,”“characters,” “terms,” “numbers,” “numerals,” or the like. These words,however, are merely convenient labels and are to be associated withappropriate physical quantities.

Unless specifically stated otherwise, discussions herein using wordssuch as “processing,” “computing,” “calculating,” “determining,”“presenting,” “displaying,” or the like may refer to actions orprocesses of a machine (e.g., a computer) that manipulates or transformsdata represented as physical (e.g., electronic, magnetic, or optical)quantities within one or more memories (e.g., volatile memory,non-volatile memory, or any suitable combination thereof), registers, orother machine components that receive, store, transmit, or displayinformation. Furthermore, unless specifically stated otherwise, theterms “a” or “an” are herein used, as is common in patent documents, toinclude one or more than one instance. Finally, as used herein, theconjunction “or” refers to a non-exclusive “or,” unless specificallystated otherwise.

Of course, the functions described herein for any single machine,database, or device may be subdivided among multiple machines,databases, or devices. As used herein, a “database” is a data storageresource and may store data structured as a text file, a table, aspreadsheet, a relational database (e.g., an object-relationaldatabase), a triple store, a hierarchical data store, or any suitablecombination thereof.

1. A method, comprising: accessing image data for an image sequence of asubject scene, the image sequence comprising a plurality of imagescaptured at different respective times, the image sequence capturingrelative movement in the subject scene between a foreground feature anda background during a time span of the image sequence; selecting fromthe image sequence a reference image that comprises a foreground portionand a background portion, the foreground portion capturing theforeground feature; in an automated operation using one or moreprocessors, deriving from the image data spatially varying blurparameters for at least part of the background portion; and generating amodified image by blurring the background portion of the reference imagebased on the spatially varying blur parameters, to simulate the relativemovement of the background during the image sequence time span, themodified image having a substantially unblurred foreground portionrelative to the reference image.
 2. The method of claim 1, furthercomprising generating per-pixel blur kernels for the reference image,based at least in part on the derived spatially varying blur parameters,and wherein the generating of the modified image comprises convolvingthe reference image and the per-pixel blur kernels.
 3. The method ofclaim 2, wherein the generating of the per-pixel blur kernels for thereference image comprises associating dotted blur kernels withrespective image pixels in the foreground feature.
 4. The method ofclaim 1, wherein the deriving of the spatially varying blur parameterscomprises: positioning a plurality of virtual cameras at respectivepositions in a reconstructed three-dimensional space of the subjectscene; projecting multiple background points in the reconstructedthree-dimensional space on to the plurality of virtual cameras, toderive a two-dimensional projection point on each of the plurality ofvirtual cameras for each of the multiple background points, so that eachof the multiple background points provides a corresponding set ofprojection points on the plurality of virtual cameras; and deriving aseparate blur kernel for each of the multiple background points based onthe corresponding set of projection points on the plurality of virtualcameras, to generate a set of spatially varying blur kernels.
 5. Themethod of claim 4, wherein the multiple background points together forma sparse subset of image pixels in the background portion, the methodfurther comprising, based on the set of spatially varying blur kernels,calculating a respective blur kernel for each background portion imagepixel that is not represented in the multiple background points, toprovide multiple blur kernels for the background portion at per-pixeldensity.
 6. The method of claim 4, wherein the deriving of the spatiallyvarying blur parameters further comprises determining a foregroundtrajectory that reflects movement of the foreground feature in thereconstructed three-dimensional space, and wherein the plurality ofvirtual cameras are positioned at different trajectory points along theforeground trajectory.
 7. The method of claim 6, wherein the pluralityof virtual cameras have a fixed, common orientation in the reconstructedthree-dimensional space, so that there is no rotation from one virtualcamera to another.
 8. The method of claim 1, further comprisingautomatically segmenting the foreground portion and the backgroundportion.
 9. The method of claim 8, wherein the segmenting of theforeground portion and the background portion comprises: determiningoptical flow information that indicates optical flow between each pairof neighboring images in the image sequence; and extracting theforeground portion based on the optical flow information.
 10. The methodof claim 1, wherein the deriving of the spatially varying blurparameters comprises reconstructing a three-dimensional space of thesubject scene.
 11. The method of claim 10, wherein the reconstructing ofthe three dimensional space comprises: tracking movement over the timespan of the image sequence of multiple background features forming partof the background portion, to determine respective feature tracks forthe multiple background features; and estimating relative depthinformation for the multiple background points in the background portionbased on the respective feature tracks.
 12. The method of claim 11,further comprising performing video stabilization of the image sequencefor tracking the movement of the multiple background features.
 13. Themethod of claim 12, wherein the performing of the video stabilizationcomprises substantially removing camera rotation for the image sequence.14. A system, comprising: an image data module that is configured toaccess image data for an image sequence of a subject scene, the imagesequence comprising a plurality of images captured at differentrespective times, the image sequence capturing relative movement in thesubject scene between a foreground feature and a background during atime span of the image sequence; a reference image data moduleconfigured to select from the image sequence a reference image thatcomprises a foreground portion and a background portion, the foregroundportion capturing the foreground feature; a blur derivation moduleconfigured to derive spatially varying blur parameters for at least partof the background portion from the image data in a substantiallyautomated operation using one or more processors; and a simulationgenerator module configured to generate a modified image by blurring thebackground portion of the reference image based on the spatially varyingblur parameters, to simulate the relative movement of the backgroundduring the image sequence time span, the modified image having asubstantially unblurred foreground portion relative to the referenceimage.
 15. The system of claim 14, wherein the blur derivation module isconfigured to derive a set of spatially varying blur kernels by derivinga separate respective blur kernel for each one of multiple backgroundpoints in the background portion of the reference image, and wherein thesimulation generator module is configured to convolve the referenceimage and the set of spatially varying blur kernels.
 16. The system ofclaim 14, further comprising: a virtual camera module configured toposition a plurality of virtual cameras at respective positions in areconstructed three-dimensional space of the subject scene; a backgroundprojection module configured to project multiple background points inthe reconstructed three-dimensional space on to the plurality of virtualcameras, to derive a two-dimensional projection point on each of theplurality of virtual cameras for each of the multiple background points,so that each of the multiple background points provides a correspondingset of projection points on the plurality of virtual cameras; and a blurkernel module configured to derive a separate blur kernel for each ofthe multiple background points based on the corresponding set ofprojection points on the plurality of virtual cameras, thereby togenerate a set of spatially varying blur kernels.
 17. The system ofclaim 16, further comprising a motion analysis module configured todetermine from the image data a foreground trajectory that reflectsmovement of the foreground feature in the reconstructed threedimensional space, wherein the plurality of virtual cameras arepositioned at different trajectory points along the foregroundtrajectory.
 18. The system of claim 14, further comprising asegmentation module configured for processing the image data toautomatically identify and segment the foreground portion and thebackground portion.
 19. The system of claim 14, further comprising amotion analysis module configured for processing the image data toreconstruct a three-dimensional space of the subject scene. 20.(canceled)
 21. A non-transitory computer-readable storage mediumincluding instructions, when executed by a computer, cause the computerto perform operations comprising: accessing image data for an imagesequence of a subject scene, the image sequence comprising a pluralityof images captured at different respective times, the image sequencecapturing relative movement in the subject scene between a foregroundfeature and a background during a time span of the image sequence;selecting from the image sequence a reference image that comprises aforeground portion and a background portion, the foreground portioncapturing the foreground feature; in an automated operation using one ormore processors, deriving spatially varying blur parameters for at leastpart of the background portion from the image data; and generating amodified image by blurring the background portion of the reference imagebased on the spatially varying blur parameters, to simulate the relativemovement of the background during the image sequence time span, themodified image having a substantially unblurred foreground portionrelative to the reference image.